From Word Types to Tokens and Back: A Survey of Approaches to Word Meaning Representation and Interpretation

Abstract Word representation paradigms situate lexical meaning at different levels of abstraction. Distributional and static embedding models generate a single vector per word type, which is an aggregate across the instances in corpus. Contextual language models, on contrary, directly capture individual instances. The goal this survey to provide overview methods, strategies that have been proposed for improving quality generated vectors. These often involve injecting external knowledge about semantic relationships, or refining vectors describe senses. also covers recent approaches obtaining type-level representations from token-level ones, combining contextualized representations. Special focus given probing interpretation studies aimed discovering encoded challenges posed by exploration motivated interest towards derivation embeddings, methods similarity estimates can be drawn space contextual models.